Repository navigation
Run Laya decisions fully in-process on ONNX Runtime - #55
Merged
Merged
Conversation
.NET 8 reaches end of support in November 2026, so the net8.0;net10.0 multi-targeting is dropped ahead of it. Every library, test, sample and benchmark project now single-targets net10.0. This is a prerequisite for consuming ElBruno.AI.Jev (a .NET 10 only package) when adding local Laya decision support, since CI previously forced the whole solution to build as net8.0. - csproj: net8.0 / net8.0;net10.0 -> net10.0 (netstandard2.0 native packaging projects unchanged) - global.json: SDK 10.0.0 - ci.yml / publish.yml / publish-bitnet-native.yml: install 10.0.x only and drop the per-framework overrides - PlaceholderIntegrationTests: drop the hardcoded -p:TargetFrameworks=net8.0 from its dotnet pack call, which re-restored the library obj/ as net8.0 and broke every later net10.0 build - docs, scripts and conventions updated to match Verified: dotnet build (0 errors) and dotnet test (1953 passed). The one remaining failure, MagenticUIServer.Agents CodeExecutorTool WSL test, is environmental (local WSL mounts read-only) and unrelated; it early-returns on the Linux CI runner. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c71e4170-a73a-4ac2-8c59-4b80834e7968
Adds a new package that runs a local decision model returning typed choice/score/yes-no answers with full probability distributions in a single forward pass, rather than generating text. Backed by Laya (Apache-2.0) over the Jev System One wire protocol, reached through ElBruno.AI.Jev 0.6.0. The wire contract was validated end to end against a live laya-serve instance: all three answer types round trip, the request omits `model` so Laya auto-routes, and extra Laya fields survive in RawRepresentation. API: - IDecisionClient with ChooseAsync / ScoreAsync / AskAsync, plus EvaluateAsync for batching several questions into one forward pass. - DecisionRequest builder (immutable) and DecisionResult reader. - ChoiceResult.ChoiceOrNull(threshold) so ambiguous inputs can fall through to a human instead of being confidently misrouted. - ProbabilityResult exposes the probability, not just a boolean, and AtThreshold re-thresholds without re-running inference. - AddLocalDecisions() DI registration, and an IJevDecisionClient constructor overload for testing without a server. DecisionOptions.Endpoint is restricted to loopback so prompts cannot leave the machine through a configuration mistake. Laya's public checkpoints report uncalibrated confidence, and Laya itself warns about out-of-range temperatures at startup. The guide, README and sample all state plainly that DecisionThreshold must be fitted on the user's own labelled data. Also registers the package in Directory.Build.props sibling versioning and adds its pack line to publish.yml. 46 new tests. Full solution: 1998 passing. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c71e4170-a73a-4ac2-8c59-4b80834e7968
The Decisions package required a running `laya-serve` Python process reached over loopback HTTP through the ElBruno.AI.Jev SDK. That put a Python install, a second process and a port in the path of every decision, which is a poor fit for a library whose whole premise is local in-process inference. Replace the transport with pure managed .NET: Microsoft.ML.OnnxRuntime for the forward pass and Microsoft.ML.Tokenizers for the input. `LayaDecisionClient` becomes `LayaOnnxDecisionClient`. The package was never published, so no consumer is broken. Reproducing Laya's numbers required porting four pieces exactly, since the model scores the hidden state at each `[MASK]` marker and a single extra or missing token shifts every later marker and silently changes the answer: - `LayaTokenizer` rebuilds the checkpoint's ByteLevel BPE from its own `tokenizer.json`. `BpeOptions.ByteLevel` only does byte<->unicode mapping and does not pre-tokenize, so `ByteLevelPreTokenizer` supplies the GPT-2 split; without it leading spaces vanish. HuggingFace also matches added tokens before BPE runs, including `[MASK]`'s `lstrip`, so that pre-pass is ported too. Validated at 22/22 against transformers.js. - `LayaSequenceBuilder` ports `build_sequence` and `render_options`, keeping Laya's budget order: options, then instructions, then the state. - `LayaRuntimeConfig` reads the fitted temperatures and clamps them into Laya's own 0.5-5.0 range. The English checkpoint ships `choice:11+ = 0.1006`, which sharpens the logits roughly tenfold and would report a 24% answer as 99% confident. - `DecisionMath` divides by temperature then softmaxes, shifting by the maximum so a large logit over a small temperature cannot overflow to NaN. Verified end-to-end: `billing 0.9696 / technical 0.0152 / sales 0.0152`, matching the reference checkpoint's published model card exactly, and the sample routes all three tickets correctly with four batched questions each. DecisionOptions drops Endpoint/ApiKey/Timeout/Model for ModelRepository, ModelPath, CacheDirectory and IntraOpNumThreads, so the community ONNX export used is fully configurable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c71e4170-a73a-4ac2-8c59-4b80834e7968
The default model source was inferenceprince/laya-onnx, a third-party repository that could be renamed or deleted out from under consumers. elbruno/laya-onnx now mirrors that export, verified byte-identical: all six files match by git/LFS OID, and inference over identical inputs produces a zero delta against the original. The weights are unmodified fp16 and Apache-2.0, with attribution to the Laya authors and the original exporter recorded in the model card. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c71e4170-a73a-4ac2-8c59-4b80834e7968
squad-ci.yml shipped as an unfilled scaffold: it echoed "No build commands configured" and exited 0 in four seconds, so it reported a passing status check while testing nothing. It fired on PRs to main, which meant every PR showed a green check that carried no signal. It now runs the same restore/build/test the CI Build workflow runs, including -p:NoBuild=false, which the BitNet native projects need or they are silently skipped. main is dropped from its triggers because ci.yml already covers main and running the suite twice on one commit buys nothing. dev, preview and insider do not exist yet, so this workflow is dormant until the dev-first branches are created — at which point they get real CI instead of none. Validated with actionlint. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c71e4170-a73a-4ac2-8c59-4b80834e7968
Both described the weights as simply coming from HuggingFace without saying which repository, so neither reflected the move to the owner-controlled mirror. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c71e4170-a73a-4ac2-8c59-4b80834e7968
…veats The choice:11+ temperature clamp was tracked with an internal, model-wide flag that callers could not read. A checkpoint can mix valid and invalid buckets, so the actionable question is whether a given answer's own bucket was clamped. Track clamping per temperature entry and surface it as CalibrationClamped on ChoiceResult, ScoreResult and ProbabilityResult. Running this against the English checkpoint shows the clamp is a guardrail, not a fix: a 12-option choice still reports 100.0% confidence, because clamping to 0.5 bounds the tenfold sharpening but any temperature below 1.0 still sharpens. The docs previously claimed clamping "keeps the number honest", which is wrong. Correct that, and state plainly that ranking is reliable while the magnitude is not. Also document fp16 batch variance, measured at 1.2e-05 to 1.6e-05 between a question asked alone and the same question asked inside a batch of four. Add a DecisionCalibration sample that demonstrates both effects with live numbers, and register it in the solution. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c71e4170-a73a-4ac2-8c59-4b80834e7968
The Pack step packs ElBruno.LocalLLMs.Decisions with --no-build, but the Build step never built it and nothing else references it, so the Release output never existed. Reproduced against a clean output directory: pack fails with NU5026, 'ElBruno.LocalLLMs.Decisions.dll to be packed was not found on disk'. Any release published from this workflow would have failed at the Pack step. Build the project alongside the other packable ones and run its test project in the Unit Tests step so the package cannot ship untested. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c71e4170-a73a-4ac2-8c59-4b80834e7968
…lish Promote the live Unreleased section to 0.22.0, matching the existing dated heading convention. Also run ElBruno.LocalLLMs.BlazorComponents.Tests in the publish workflow. The package was packed and pushed without its 23 tests ever running in the release gate. Verified passing under the same Release configuration the gate uses. Two stale Unreleased sections remain further down the file, describing features that shipped long ago. They predate this work and are left untouched. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c71e4170-a73a-4ac2-8c59-4b80834e7968
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
ElBruno.LocalLLMs.Decisionsshipped requiring a runninglaya-servePython process, reached over loopback HTTP through theElBruno.AI.JevSDK. That put a Python install, a second process and a port in the path of every decision — a poor fit for a library whose whole premise is local, in-process inference.This replaces that transport with pure managed .NET:
Microsoft.ML.OnnxRuntimefor the forward pass andMicrosoft.ML.Tokenizersfor the input. No server, no Python, no network call at inference time.LayaDecisionClientbecomesLayaOnnxDecisionClient. The package was never published to NuGet, so no consumer is broken.What made this non-trivial
The model scores the hidden state at each
[MASK]marker, so a single extra or missing token shifts every later marker and silently changes the answer rather than failing. Four pieces had to be ported exactly:LayaTokenizer+ByteLevelPreTokenizertokenizer.json.BpeOptions.ByteLevelonly does byte↔unicode mapping and does not pre-tokenize, so the GPT-2 regex split is supplied separately — without it leading spaces vanish. HuggingFace also matches added tokens before BPE runs, including[MASK]'slstrip, so that pre-pass is ported too.LayaSequenceBuilderbuild_sequence/render_options, preserving Laya's budget order: options, then instructions, then the state.LayaRuntimeConfigDecisionMathThe tokenizer was the main risk. It was iterated to 22/22 exact parity against a
transformers.jsoracle covering Laya prompt fragments, Unicode, emoji, CJK, Cyrillic, whitespace runs and a literal[MASK].Judgement call worth reviewing
Laya's English checkpoint ships
choice:11+ = 0.1006, far below its ownTEMP_MINof0.5. That sharpens the logits roughly tenfold. This clamps it into Laya's valid range and reports it per answer viaCalibrationClampedonChoiceResult,ScoreResultandProbabilityResult— per answer rather than model-wide, because a checkpoint can mix valid and invalid buckets.The clamp is a guardrail, not a fix. Measured on the real checkpoint, a 12-option choice still reports 100.0% confidence after clamping, because any temperature below
1.0still sharpens. So the ranking stays reliable while the magnitude does not, andChoiceOrNullwill never abstain in that bucket. The guide and the newDecisionCalibrationsample both say this outright.Verification
Every path was exercised against the real 800 MB model, not just compiled:
billing 0.9696 / technical 0.0152 / sales 0.0152, matching the reference checkpoint's published model card..nupkg, not a project reference — proving ONNX Runtime's native assets flow to consumers.model.onnx.dataexternal weights the graph is unusable without.DecisionCalibrationsample prints this live.MagenticUIServer(confirmed failing on a clean tree without these changes).API change
DecisionOptionsdropsEndpoint/ApiKey/Timeout/Modelin favour ofModelRepository,ModelPath,CacheDirectoryandIntraOpNumThreads.Laya's authors publish PyTorch weights only, so every ONNX export of Laya is community-produced. The default is now
elbruno/laya-onnx, an unmodified mirror ofinferenceprince/laya-onnxpublished so this library does not depend on a third party's repo staying available. It was verified identical two ways: every file matches by git/LFS OID, and inference on identical inputs differs by0.00E+000. The source stays fully configurable so consumers can pin an export they control.Docs, README, samples and CHANGELOG updated; version bumped to 0.22.0. Adds a
DecisionCalibrationsample that demonstrates the clamp and the fp16 batch variance with live numbers.